System and method for pose estimation of an imaging device and for determining the location of a medical device with respect to a target

ABSTRACT

A system and method for constructing fluoroscopic-based three-dimensional volumetric data of a target area within a patient from two-dimensional fluoroscopic images acquired via a fluoroscopic imaging device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of the filing date of provisionalU.S. Patent Application No. 62/628,017, filed Feb. 8, 2018, provisionalU.S. Patent Application No. 62/641,777, filed Mar. 12, 2018, and U.S.patent application Ser. No. 16/022,222, filed Jun. 28, 2018, the entirecontents of each of which are incorporated herein by reference.

BACKGROUND

The disclosure relates to the field of imaging, and particularly to theestimation of a pose of an imaging device and to three-dimensionalimaging of body organs.

Pose estimation of an imaging device, such as a camera or a fluoroscopicdevice, may be required or used for variety of applications, includingregistration between different imaging modalities or the generation ofaugmented reality. One of the known uses of a pose estimation of animaging device is the construction of a three-dimensional volume from aset of two-dimensional images captured by the imaging device while indifferent poses. Such three-dimensional construction is commonly used inthe medical field and has a significant impact.

There are several commonly applied medical methods, such as endoscopicprocedures or minimally invasive procedures, for treating variousmaladies affecting organs including the liver, brain, heart, lung, gallbladder, kidney and bones. Often, one or more imaging modalities, suchas magnetic resonance imaging, ultrasound imaging, computed tomography(CT), fluoroscopy as well as others are employed by clinicians toidentify and navigate to areas of interest within a patient andultimately targets for treatment. In some procedures, pre-operativescans may be utilized for target identification and intraoperativeguidance. However, real-time imaging may be often required in order toobtain a more accurate and current image of the target area.Furthermore, real-time image data displaying the current location of amedical device with respect to the target and its surrounding may berequired in order to navigate the medical device to the target in a moresafe and accurate manner (e.g., with unnecessary or no damage caused toother tissues and organs).

SUMMARY

According to one aspect of the disclosure, a system for constructingfluoroscopic-based three-dimensional volumetric data of a target areawithin a patient from two-dimensional fluoroscopic images acquired via afluoroscopic imaging device is provided. The system includes a structureof markers and a computing device. A sequence of images of the targetarea and of the structure of markers is acquired via the fluoroscopicimaging device. The computing device is configured to estimate a pose ofthe fluoroscopic imaging device for a plurality of images of thesequence of images based on detection of a possible and most probableprojection of the structure of markers as a whole on each image of theplurality of images, and construct fluoroscopic-based three-dimensionalvolumetric data of the target area based on the estimated poses of thefluoroscopic imaging device.

In an aspect, the computing device is further configured to facilitatean approach of a medical device to the target area, wherein a medicaldevice is positioned in the target area prior to acquiring the sequenceof images, and determine an offset between the medical device and thetarget based on the fluoroscopic-based three-dimensional volumetricdata.

In an aspect, the system further comprises a locating system indicatinga location of the medical device within the patient. Additionally, thecomputing device may be further configured to display the target areaand the location of the medical device with respect to the target,facilitate navigation of the medical device to the target area via thelocating system and the display, and correct the display of the locationof the medical device with respect to the target based on the determinedoffset between the medical device and the target.

In an aspect, the computing device is further configured to display a 3Drendering of the target area on the display, and register the locatingsystem to the 3D rendering, wherein correcting the display of thelocation of the medical device with respect to the target comprisesupdating the registration between the locating system and the 3Drendering.

In an aspect, the locating system is an electromagnetic locating system.

In an aspect, the target area comprises at least a portion of lungs andthe medical device is navigable to the target area through airways of aluminal network.

In an aspect, the structure of markers is at least one of a periodicpattern or a two-dimensional pattern. The target area may include atleast a portion of lungs and the target may be a soft tissue target.

In yet another aspect of the disclosure, a method for constructingfluoroscopic-based three dimensional volumetric data of a target areawithin a patient from a sequence of two-dimensional (2D) fluoroscopicimages of a target area and of a structure of markers acquired via afluoroscopic imaging device is provided. The structure of markers ispositioned between the patient and the fluoroscopic imaging device. Themethod includes using at least one hardware processor for estimating apose of the fluoroscopic imaging device for at least a plurality ofimages of the sequence of 2D fluoroscopic images based on detection of apossible and most probable projection of the structure of markers as awhole on each image of the plurality of images, and constructingfluoroscopic-based three-dimensional volumetric data of the target areabased on the estimated poses of the fluoroscopic imaging device.

In an aspect, a medical device is positioned in the target area prior toacquiring the sequence of images, and wherein the method furthercomprises using the at least one hardware processor for determining anoffset between the medical device and the target based on thefluoroscopic-based three-dimensional volumetric data.

In an aspect, the method further includes facilitating navigation of themedical device to the target area via a locating system indicating alocation of the medical device and via a display, and correcting adisplay of the location of the medical device with respect to the targetbased on the determined offset between the medical device and thetarget.

In an aspect, the method further includes displaying a 3D rendering ofthe target area on the display, and registering the locating system tothe 3D rendering, where the correcting of the location of the medicaldevice with respect to the target comprises updating the registration ofthe locating system to the 3D rendering.

In an aspect, the method further includes using the at least onehardware processor for generating the 3D rendering of the target areabased on previously acquired CT volumetric data of the target area.

In an aspect, the target area includes at least a portion of lungs andthe medical device is navigable to the target area through airways of aluminal network.

In an aspect, the structure of markers is at least one of a periodicpattern or a two-dimensional pattern. The target area may include atleast a portion of lungs and the target may be a soft-tissue target.

In yet another aspect of the disclosure, a system for constructingfluoroscopic-based three-dimensional volumetric data of a target areawithin a patient from two-dimensional fluoroscopic images acquired via afluoroscopic imaging device is provided. The system includes a computingdevice configured to estimate a pose of the fluoroscopic imaging devicefor a plurality of images of a sequence of images based on detection ofa possible and most probable projection of a structure of markers as awhole on each image of the plurality of images, and constructfluoroscopic-based three-dimensional volumetric data of the target areabased on the estimated poses of the fluoroscopic imaging device.

In an aspect, the computing device is further configured to facilitatean approach of a medical device to the target area, wherein a medicaldevice is positioned in the target area prior to acquisition of thesequence of images, and determine an offset between the medical deviceand the target based on the fluoroscopic-based three-dimensionalvolumetric data.

In an aspect, the computing device is further configured to display thetarget area and the location of the medical device with respect to thetarget, facilitate navigation of the medical device to the target areavia the locating system and the display, and correct the display of thelocation of the medical device with respect to the target based on thedetermined offset between the medical device and the target.

In an aspect, the computing device is further configured to display a 3Drendering of the target area on the display, and register the locatingsystem to the 3D rendering, wherein correcting the display of thelocation of the medical device with respect to the target comprisesupdating the registration between the locating system and the 3Drendering.

BRIEF DESCRIPTION OF THE DRAWINGS

Various exemplary embodiments are illustrated in the accompanyingfigures with the intent that these examples not be restrictive. It willbe appreciated that for simplicity and clarity of the illustration,elements shown in the figures referenced below are not necessarily drawnto scale. Also, where considered appropriate, reference numerals may berepeated among the figures to indicate like, corresponding or analogouselements. The figures are listed below.

FIG. 1 is a flow chart of a method for estimating the pose of an imagingdevice by utilizing a structure of markers in accordance with one aspectof the disclosure;

FIG. 2A is a schematic diagram of a system configured for use with themethod of FIG. 1 in accordance with one aspect of the disclosure;

FIG. 2B is a schematic illustration of a two-dimensional grid structureof sphere markers in accordance with one aspect of the disclosure;

FIG. 3 shows an exemplary image captured by a fluoroscopic device of anartificial chest volume of a Multipurpose Chest Phantom N1 “LUNGMAN”, byKyoto Kagaku, placed over the grid structure of radio-opaque markers ofFIG. 2B;

FIG. 4 is a probability map generated for the image of FIG. 3 inaccordance with one aspect of the disclosure;

FIGS. 5A-5C show different exemplary candidates for the projection ofthe 2D grid structure of sphere markers of FIG. 2B on the image of FIG.3 overlaid on the probability map of FIG. 4;

FIG. 6A shows a selected candidate for the projection of the 2D gridstructure of sphere markers of FIG. 2B on the image of FIG. 3, overlaidon the probability map of FIG. 4 in accordance with one aspect of thedisclosure;

FIG. 6B shows an improved candidate for the projection of the 2D gridstructure of sphere markers of FIG. 2B on the image of FIG. 3, overlaidon the probability map of FIG. 4 in accordance with one aspect of thedisclosure;

FIG. 6C shows a further improved candidate for the projection of the 2Dgrid structure of sphere markers of FIG. 2B on image 300 of FIG. 3,overlaid on the probability map of FIG. 4 in accordance with one aspectof the disclosure;

FIG. 7 is a flow chart of an exemplary method for constructingfluoroscopic three-dimensional volumetric data in accordance with oneaspect of the disclosure; and

FIG. 8 is a view of one illustrative embodiment of an exemplary systemfor constructing fluoroscopic-based three-dimensional volumetric data inaccordance with the disclosure.

DETAILED DESCRIPTION

Prior art methods and systems for pose estimation may be inappropriatefor real time use, inaccurate or non-robust. Therefore, there is a needfor a method and system, which provide a relatively fast, accurate androbust pose estimation, particularly in the field of medical imaging.

In order to navigate medical devices to a remote target for example, forbiopsy or treatment, both the medical device and the target should bevisible in some sort of a three-dimensional guidance system. When thetarget is a small soft-tissue object, such as a tumor or a lesion, anX-ray volumetric reconstruction is needed in order to be able toidentify it. Several solutions exist that provide three-dimensionalvolume reconstruction such as CT and Cone-beam CT which are extensivelyused in the medical world. These machines algorithmically combinemultiple X-ray projections from known, calibrated X-ray source positionsinto three dimensional volume in which, inter alia, soft-tissues arevisible. For example, a CT machine can be used with iterative scansduring procedure to provide guidance through the body until the toolsreach the target. This is a tedious procedure as it requires severalfull CT scans, a dedicated CT room and blind navigation between scans.In addition, each scan requires the staff to leave the room due tohigh-levels of ionizing radiation and exposes the patient to suchradiation. Another option is a Cone-beam CT machine which is availablein some operation rooms and is somewhat easier to operate, but isexpensive and like the CT only provides blind navigation between scans,requires multiple iterations for navigation and requires the staff toleave the room. In addition, a CT-based imaging system is extremelycostly, and in many cases not available in the same location as thelocation where a procedure is carried out.

A fluoroscopic imaging device is commonly located in the operating roomduring navigation procedures. The standard fluoroscopic imaging devicemay be used by a clinician, for example, to visualize and confirm theplacement of a medical device after it has been navigated to a desiredlocation. However, although standard fluoroscopic images display highlydense objects such as metal tools and bones as well as large soft-tissueobjects such as the heart, the fluoroscopic images have difficultyresolving small soft-tissue objects of interest such as lesions.Furthermore, the fluoroscope image is only a two-dimensional projection,while in order to accurately and safely navigate within the body, avolumetric or three-dimensional imaging is required.

An endoscopic approach has proven useful in navigating to areas ofinterest within a patient, and particularly so for areas within luminalnetworks of the body such as the lungs. To enable the endoscopic, andmore particularly the bronchoscopic, approach in the lungs,endobronchial navigation systems have been developed that use previouslyacquired MRI data or CT image data to generate a three dimensionalrendering or volume of the particular body part such as the lungs.

The resulting volume generated from the MRI scan or CT scan is thenutilized to create a navigation plan to facilitate the advancement of anavigation catheter (or other suitable medical device) through abronchoscope and a branch of the bronchus of a patient to an area ofinterest. A locating system, such as an electromagnetic tracking system,may be utilized in conjunction with the CT data to facilitate guidanceof the navigation catheter through the branch of the bronchus to thearea of interest. In certain instances, the navigation catheter may bepositioned within one of the airways of the branched luminal networksadjacent to, or within, the area of interest to provide access for oneor more medical instruments.

As another example, minimally invasive procedures, such as laparoscopyprocedures, including robotic-assisted surgery, may employintraoperative fluoroscopy in order to increase visualization, e.g., forguidance and lesion locating, or in order to prevents injury andcomplications.

Therefore, a fast, accurate and robust three-dimensional reconstructionof images is required, which is generated based on a standardfluoroscopic imaging performed during medical procedures.

FIG. 1 illustrates a flow chart of a method for estimating the pose ofan imaging device by utilizing a structure of markers in accordance withan aspect of the disclosure. In a step 100, a probability map may begenerated for an image captured by an imaging device. The image includesa projection of a structure of markers. The probability map may indicatethe probability of each pixel of the image to belong to the projectionof a marker of the structure of markers. In some embodiments, thestructure of markers may be of a two-dimensional pattern. In someembodiments, the structure of markers may be of a periodic pattern, suchas a grid. The image may include a projection of at least a portion ofthe structure of markers.

Reference is now made to FIGS. 2B and 3. FIG. 2B is a schematicillustration of a two-dimensional (2D) grid structure of sphere markers220 in accordance with the disclosure. FIG. 3 is an exemplary image 300captured by a fluoroscopic device of an artificial chest volume of aMultipurpose Chest Phantom N1 “LUNGMAN”, by Kyoto Kagaku, placed overthe 2D grid structure of sphere markers 220 of FIG. 2B. 2D gridstructure of sphere markers 220 includes a plurality of sphere shapedmarkers, such as sphere markers 230 a and 230 b, arranged in atwo-dimensional grid pattern. Image 300 includes a projection of aportion of 2D grid structure of sphere markers 220 and a projection of acatheter 320. The projection of 2D grid structure of sphere markers 220on image 300 includes projections of the sphere markers, such as spheremarker projections 310 a, 310 b and 310 c.

The probability map may be generated, for example, by feeding the imageinto a simple marker (blob) detector, such as a Harris corner detector,which outputs a new image of smooth densities, corresponding to theprobability of each pixel to belong to a marker. FIG. 4 illustrates aprobability map 400 generated for image 300 of FIG. 3. Probability map400 includes pixels or densities, such as densities 410 a, 410 b and 410c, which correspond accordingly to markers 310 a, 310 b and 310 c. Insome embodiments, the probability map may be downscaled (e.g., reducedin size) in order to make the required computations more simple andefficient. It should be noted that probability map 400, as shown inFIGS. 5A-6B is downscaled by four and probability map 400 as shown inFIG. 6C is downscaled by two.

In a step 110, different candidates may be generated for the projectionof the structure of markers on the image. The different candidates maybe generated by virtually positioning the imaging device in a range ofdifferent possible poses. By “possible poses” of the imaging device, itis meant three-dimensional positions and orientations of the imagingdevice. In some embodiments, such a range may be limited according tothe geometrical structure and/or degrees of freedom of the imagingdevice. For each such possible pose, a virtual projection of at least aportion of the structure of markers is generated, as if the imagingdevice actually captured an image of the structure of markers whilepositioned at that pose.

In a step 120, the candidate having the highest probability of being theprojection of the structure of markers on the image may be identifiedbased on the image probability map. Each candidate, e.g., a virtualprojection of the structure of markers, may be overlaid or associated tothe probability map. A probability score may be then determined orassociated with each marker projection of the candidate. In someembodiments, the probability score may be positive or negative, e.g.,there may be a cost in case virtual markers projections falls withinpixels of low probability. The probability scores of all of the markersprojections of a candidate may be then summed and a total probabilityscore may be determined for each candidate. For example, if thestructure of markers is a two-dimensional grid, then the projection willhave a grid form. Each point of the projection grid would lie on atleast one pixel of the probability map. A 2D grid candidate will receivethe highest probability score if its points lie on the highest densitypixels, that is, if its points lie on projections of the centeres of themarkers on the image. The candidate having the highest probability scoremay be determined as the candidate which has the highest probability ofbeing the projection of the structure of markers on the image. The poseof the imaging device for the image may be then estimated based on thevirtual pose of the imaging device used to generate the identifiedcandidate.

FIGS. 5A-5C illustrate different exemplary candidates 500 a-c for theprojection of 2D grid structure of sphere markers 220 of FIG. 2B onimage 300 of FIG. 3 overlaid on probability map 400 of FIG. 4.Candidates 500 a, 500 b and 500 c are indicated as a grid of plus signs(“+”), while each such sign indicates the center of a projection of amarker. Candidates 500 a, 500 b and 500 c are virtual projections of 2Dgrid structure of sphere markers 220, as if the fluoroscope used tocapture image 300 is located at three different poses associatedcorrespondingly with these projections. In this example, candidate 500 awas generated as if the fluoroscope is located at: position [0, −50, 0],angle: −20 degrees. Candidate 500 b was generated as if the fluoroscopeis located at: position [0, −10, 0], angle: −20 degrees. Candidate 500 cwas generated as if the fluoroscope is located at: position [7.5, −40,11.25], angle: −25 degrees. The above-mentioned coordinates are withrespect to 2D grid structure of sphere markers 220. Densities 410 a ofprobability map 400 are indicated in FIGS. 5A-5C. Plus signs 510 a, 510b and 510 c are the centers of the markers projections of candidates 500a, 500 b and 500 c correspondingly, which are the ones closest todensities 410 a. One can see that plus sign 510 c is the sign which bestfits densities 410 a and therefore would receive the highest probabilityscore among signs 510 a, 510 b and 510 c of candidates 500 a, 500 b and500 c correspondingly. One can further see that accordingly, candidate500 c would receive the highest probability score since its markersprojections best fit probability map 400. Thus, among these threeexemplary candidates, 500 a, 500 b and 500 c, candidate 500 c would beidentified as the candidate with the highest probability of being theprojection of 2D grid structure of sphere markers 220 on image 300.

Further steps may be performed in order to refine the above describedpose estimation. In an optional step 130, a locally deformed version ofthe candidate may be generated in order to maximize its probability ofbeing the projection of the structure of markers on the image. Thelocally deformed version may be generated based on the image probabilitymap. A local search algorithm may be utilized to deform the candidate sothat it would maximize its score. For example, in case the structure ofmarkers is a 2D grid, each 2D grid point may be treated individually.Each point may be moved towards the neighbouring local maxima on theprobability map using gradient ascent method.

In an optional step 140, an improved candidate for the projection of thestructure of markers on the image may be detected based on the locallydeformed version of the candidate. The improved candidate is determinedsuch that it fits (exactly or approximately) the locally deformedversion of the candidate. Such improved candidate may be determined byidentifying a transformation that will fit a new candidate to the localdeformed version, e.g., by using homography estimation methods. Thevirtual pose of the imaging device associated with the improvedcandidate may be then determined as the estimated pose of the imagingdevice for the image.

In some embodiments, the generation of a locally deformed version of thecandidate and the determination of an improved candidate may beiteratively repeated. These steps may be iteratively repeated until theprocess converges to a specific virtual projection of the structure ofmarkers on the image, which may be determined as the improved candidate.Thus, since the structure of markers converges as a whole, false localmaxima is avoided. In an aspect, as an alternative to using a list ofcandidates and finding an optimal candidate for estimating the camerapose, the camera pose may be estimated by solving a homography thattransforms a 2D fiducial structure in 3D space into image coordinatesthat matches the fiducial probability map generated from the imagingdevice output.

FIG. 6A shows a selected candidate 600 a, for projection of 2D gridstructure of sphere markers 220 of FIG. 2B on image 300 of FIG. 3,overlaid on probability map 400 of FIG. 4. FIG. 6B shows an improvedcandidate 600B, for the projection of 2D grid structure of spheremarkers 220 of FIG. 2B on image 300 of FIG. 3, overlaid on probabilitymap 400 of FIG. 4. FIG. 6C shows a further improved candidate 600 c, forthe projection of 2D grid structure of sphere markers 220 of FIG. 2B onimage 300 of FIG. 3, overlaid on probability map 400 of FIG. 4. Asdescribed above, the identified or selected candidate is candidate 500c, which is now indicated 600 a. Candidate 600 b is the improvedcandidate which was generated based on a locally deformed version ofcandidate 600 a according to the method disclosed above. Candidate 600 cis a further improved candidate with respect to candidate 600 b,generated by iteratively repeating the process of locally deforming theresulting candidate and determining an approximation to maximize thecandidate probability. FIG. 6C illustrates the results of refinedcandidates based on a higher resolution probability map. In an aspect,this is done after completing a refinement step using the down-sampledversion of the probability map. Plus signs 610 a, 610 b and 610 c arethe centers of the markers projections of candidates 600 a, 600 b and600 c correspondingly, which are the ones closest to densities 410 a ofprobability map 400. One can see how the candidates for the projectionof 2D grid structure of sphere markers 220 on image 300 converge to thecandidate of the highest probability according to probability map 400.

In some embodiments, the imaging device may be configured to capture asequence of images. A sequence of images may be captured, automaticallyor manually, by continuously sweeping the imaging device at a certainangle. When pose estimation of a sequence of images is required, theestimation process may become more efficient by reducing the range orarea of possible virtual poses for the imaging device. A plurality ofnon-sequential images of the sequence of images may be then determined.For example, the first image in the sequence, the last image, and one ormore images in-between. The one or more images in-between may bedetermined such that the sequence is divided into equal image portions.At a first stage, the pose of the imaging device may be estimated onlyfor the determined non-sequential images. At a second stage, the area orrange of possible different poses for virtually positioning the imagingdevice may be reduced. The reduction may be performed based on theestimated poses of the imaging device for the determined non-sequentialimages. The pose of the imaging device for the rest of the images may bethen estimated according to the reduced area or range. For example, thepose of the imaging device for the first and tenth images of thesequence are determined at the first stage. The pose of the imagingdevice for the second to ninth images must be along a feasible andcontinuous path between its pose for the first image and its pose forthe tenth image, and so on.

In some embodiments, geometrical parameters of the imaging device may bepre-known, or pre-determined, such as the field of view of the source,height range, rotation angle range and the like, including the devicedegrees of freedom (e.g., independent motions allowed). In someembodiments, such geometrical parameters of the imaging device may bedetermined in real-time while estimating the pose of the imaging devicefor the captured images. Such information may be also used to reduce thearea or range of possible poses. In some embodiments, a user practicingthe disclosed disclosure may be instructed to limit the motion of theimaging device to certain degrees of freedom or to certain ranges ofmotion for the sequence of images. Such limitations may be alsoconsidered when determining the imaging device possible poses and thusmay be used to make the imaging device pose estimation faster.

In some embodiments, an image pre-processing methods may be firstapplied to the one or more images in order to correct distortions and/orenhance the visualization of the projection of the structure of markerson the image. For example, in case the imaging device is a fluoroscope,correction of “pincushion” distortion, which slightly warps the image,may be performed. This distortion may be automatically addressed bymodelling the warp with a polynomial surface and applying compatiblewarp which will cancel out the pincushion effect. In case a grid ofmetal spheres is used, the image may be inversed in order to enhance theprojections of the markers. In addition, the image may be blurred usingGaussian filter with sigma value equal, for example, to one half of thespheres diameter, in order to facilitate the search and evaluation ofcandidates as disclosed above.

In some embodiments, one or more models of the imaging device may becalibrated to generate calibration data, such as a data file, which maybe used to automatically calibrate the specific imaging device. Thecalibration data may include data referring to the geometric calibrationand/or distortion calibration, as disclosed above. In some embodiments,the geometric calibration may be based on data provided by the imagingdevice manufacturer. In some embodiments, a manual distortioncalibration may be performed once for a specific imaging device. In anaspect, the imaging device distortion correction can be calibrated as apreprocessing step during every procedure as the pincushion distortionmay change as a result of imaging device maintenance or even as a resultof a change in time.

FIG. 2A illustrates a schematic diagram of a system 200 configured foruse with the method of FIG. 1 in accordance with one aspect of thedisclosure. System 200 may include a workstation 80, an imaging device215 and a structure of markers structure 218. In some embodiments,workstation 80 may be coupled with imaging device 215, directly orindirectly, e.g., by wireless communication. Workstation 80 may includea memory 202, a processor 204, a display 206 and an input device 210.Processor or hardware processor 204 may include one or more hardwareprocessors. Workstation 80 may optionally include an output module 212and a network interface 208. Memory 202 may store an application 81 andimage data 214. Application 81 may include instructions executable byprocessor 204, inter alia, for executing the method of FIG. 1 and a userinterface 216. Workstation 80 may be a stationary computing device, suchas a personal computer, or a portable computing device such as a tabletcomputer. Workstation 80 may embed a plurality of computer devices.

Memory 202 may include any non-transitory computer-readable storagemedia for storing data and/or software including instructions that areexecutable by processor 204 and which control the operation ofworkstation 80 and in some embodiments, may also control the operationof imaging device 215. In an embodiment, memory 202 may include one ormore solid-state storage devices such as flash memory chips.Alternatively, or in addition to the one or more solid-state storagedevices, memory 202 may include one or more mass storage devicesconnected to the processor 204 through a mass storage controller (notshown) and a communications bus (not shown). Although the description ofcomputer-readable media contained herein refers to a solid-statestorage, it should be appreciated by those skilled in the art thatcomputer-readable storage media can be any available media that can beaccessed by the processor 204. That is, computer readable storage mediamay include non-transitory, volatile and non-volatile, removable andnon-removable media implemented in any method or technology for storageof information such as computer-readable instructions, data structures,program modules or other data. For example, computer-readable storagemedia may include RAM, ROM, EPROM, EEPROM, flash memory or othersolid-state memory technology, CD-ROM, DVD, Blu-Ray or other opticalstorage, magnetic cassettes, magnetic tape, magnetic disk storage orother magnetic storage devices, or any other medium which may be used tostore the desired information and which may be accessed by workstation80.

Application 81 may, when executed by processor 204, cause display 206 topresent user interface 216. Network interface 208 may be configured toconnect to a network such as a local area network (LAN) consisting of awired network and/or a wireless network, a wide area network (WAN), awireless mobile network, a Bluetooth network, and/or the internet.Network interface 208 may be used to connect between workstation 80 andimaging device 215. Network interface 208 may be also used to receiveimage data 214. Input device 210 may be any device by means of which auser may interact with workstation 80, such as, for example, a mouse,keyboard, foot pedal, touch screen, and/or voice interface. Outputmodule 212 may include any connectivity port or bus, such as, forexample, parallel ports, serial ports, universal serial busses (USB), orany other similar connectivity port known to those skilled in the art.

Imaging device 215 may be any imaging device, which captures 2D images,such as a standard fluoroscopic imaging device or a camera. In someembodiments, markers structure 218, may be a structure of markers havinga two-dimensional pattern, such as a grid having two dimensions of widthand length (e.g., 2D grid), as shown in FIG. 2B. Using a 2D pattern, asopposed to a 3D pattern, may facilitate the pose estimation process.Furthermore, when for example, a patient is required to lie on markersstructure 218 in order to estimate the pose of a medical imaging devicewhile scanning the patient, a 2D pattern would be more convenient forthe patient. The markers should be formed such that they will be visiblein the imaging modality used. For example, if the imaging device is afluoroscopic device, then the markers should be made of a material whichis at least partially radio-opaque. In some embodiments, the shape ofthe markers may be symmetric and such that the projection of the markerson the image would be the same at any pose the imaging device may beplaced. Such configuration may simplify and enhance the pose estimationprocess and/or make it more efficient. For example, when the imagingdevice is rotated around the markers structure, markers having arotation symmetry may be preferred, such as spheres. The size of themarkers structure and/or the number of markers in the structure may bedetermined according to the specific use of the disclosed systems andmethods. For example, if the pose estimation is used to construct a 3Dvolume of an area of interest within a patient, then the markersstructure may be of a size similar or larger than the size of the areaof interest. In some embodiments, the pattern of markers structure 218may be two-dimensional and/or periodic, such as a 2D grid. Using aperiodic and/or of a two-dimensional pattern structure of markers mayfurther enhance and facilitate the pose estimation process and make itmore efficient.

Referring now to FIG. 2B, 2D grid structure of sphere markers 220 has a2D periodic pattern of a grid and includes symmetric markers in theshape of a sphere. Such a configuration simplifies and enhances the poseestimation process, as described in FIG. 1, specifically when generatingthe virtual candidates for the markers structure projection and whendetermining the optimal one. The structure of markers, as a fiducial,should be positioned in a stationary manner during the capturing of theone or more images. In an exemplary 2D grid structure of sphere markerssuch as described above, used in medical imaging of the lungs area, thesphere markers diameter may be 2±0.2 mm and the distance between thespheres may be about 15±0.15 mm isotropic.

Referring now back to FIG. 2A, imaging device 215 may capture one ormore images (i.e., a sequence of images) such that at least a projectionof a portion of markers structure 218 is shown in each image. The imageor sequence of images captured by imaging device 215 may be then storedin memory 202 as image data 214. The image data may be then processed byprocessor 204 and according to the method of FIG. 1, to determine thepose of imaging device 215. The pose estimation data may be then outputvia output module 212, display 206 and/or network interface 208. Markersstructure 218 may be positioned with respect to an area of interest,such as under an area of interest within the body of a patient goingthrough a fluoroscopic scan. Markers structure 218 and the patient willthen be positioned such that the one or more images captured by imagingdevice 215 would capture the area of interest and a portion of markersstructure 218. If required, once the pose estimation process iscomplete, the projection of markers structure 218 on the images may beremoved by using well known methods. One such method is described incommonly-owned U.S. patent application Ser. No. 16/259,612, entitled:“IMAGE RECONSTRUCTION SYSTEM AND METHOD”, filed on Jan. 28, 2019, byAlexandroni et al., the entire content of which is hereby incorporatedby reference.

FIG. 7 is a flow chart of an exemplary method for constructingfluoroscopic three-dimensional volumetric data in accordance with thedisclosure. A method for constructing fluoroscopic-basedthree-dimensional volumetric data of a target area within a patient fromtwo dimensional fluoroscopic images, is hereby disclosed. In step 700, asequence of images of the target area and of a structure of markers isacquired via a fluoroscopic imaging device. The structure of markers maybe the two-dimensional structure of markers described with respect toFIGS. 1, 2A and 2B. The structure of markers may be positioned betweenthe patient and the fluoroscopic imaging device. In some embodiments,the target area may include, for example, at least a portion of thelungs, and as exemplified with respect to the system of FIG. 8. In someembodiments, the target is a soft-tissue target, such as within a lung,kidney, liver and the like.

In a step 710, a pose of the fluoroscopic imaging device for at least aplurality of images of the sequence of images may be estimated. The poseestimation may be performed based on detection of a possible and mostprobable projection of the structure of markers as a whole on each imageof the plurality of images, and as described with respect to FIG. 1.

In some embodiments, other methods for estimating the pose of thefluoroscopic device may be used. There are various known methods fordetermining the poses of imaging devices, such as an external anglemeasuring device or based on image analysis. Some of such devices andmethods are particularly described in commonly-owned U.S. PatentPublication No. 2017/0035379, filed on Aug. 1, 2016, by Weingarten etal, the entire content of which is hereby incorporated by reference.

In a step 720, a fluoroscopic-based three-dimensional volumetric data ofthe target area may be constructed based on the estimated poses of thefluoroscopic imaging device. Exemplary systems and methods forconstructing such fluoroscopic-based three-dimensional volumetric dataare disclosed in the above commonly-owned U.S. Patent Publication No.2017/0035379, which is incorporated by reference.

In an optional step 730, a medical device may be positioned in thetarget area prior to the acquiring of the sequence of images. Thus, thesequence of images and consequently the fluoroscopic-basedthree-dimensional volumetric data may also include a projection of themedical device in addition to the target. The offset (i.e., Δx, Δy andΔz) between the medical device and the target may be then determinedbased on the fluoroscopic-based three-dimensional volumetric data. Thetarget may be visible or better exhibited in the generatedthree-dimensional volumetric data. Therefore, the target may bedetected, automatically, or manually by the user, in thethree-dimensional volumetric data. The medical device may be detected,automatically or manually by a user, in the sequence of images, ascaptured, or in the generated three-dimensional volumetric data. Theautomatic detection of the target and/or the medical device may beperformed based on systems and methods as known in the art and such asdescribed, for example, in commonly-owned U.S. Patent Application No.62/627,911, titled: “SYSTEM AND METHOD FOR CATHETER DETECTION INFLUOROSCOPIC IMAGES AND UPDATING DISPLAYED POSITION OF CATHETER”, filedon Feb. 8, 2018, by Birenbaum et al. The manual detection may beperformed by displaying to the user the three-dimensional volumetricdata and/or captured images and requesting his input. Once the targetand the medical device are detected in the three-dimensional volumetricdata and/or the captures images, their location in the fluoroscopiccoordinate system of reference may be obtained and the offset betweenthem may be determined.

The offset between the target and the medical device may be utilized forvarious medical purposes, including facilitating approach of the medicaldevice to the target area and treatment. The navigation of a medicaldevice to the target area may be facilitated via a locating system and adisplay. The locating system locates or tracks the motion of the medicaldevice through the patient's body. The display may display the medicaldevice location to the user with respect to the surroundings of themedical device within the patient's body and the target. The locatingsystem may be, for example, an electromagnetic or optic locating system,or any other such system as known in the art. When, for example, thetarget area includes a portion of the lungs, the medical device may benavigated to the target area through the airways luminal network and asdescribed with respect to FIG. 8.

In an optional step 740, a display of the location of the medical devicewith respect to the target may be corrected based on the determinedoffset between the medical device and the target. In some embodiments, a3D rendering of the target area may be displayed on the display. The 3Drendering of the target area may be generated based on CT volumetricdata of the target area which was acquired previously, e.g., prior tothe current procedure or operation (e.g., preoperative CT). In someembodiments, the locating system may be registered to the 3D renderingof the target, such as described, for example, with respect to FIG. 8below. The correction of the offset between the medical device and thetarget may be then performed by updating the registration of thelocating system to the 3D rendering. Generally, to perform suchupdating, a transformation between coordinate system of reference of thefluoroscopic images and the coordinate system of reference of thelocating system should be known. The geometrical positioning of thestructure of markers with respect to the locating system may determinesuch a transformation. In some embodiments, and as shown in theembodiment of FIG. 8, the structure of markers and the locating systemare positioned such that the same coordinate system of reference wouldapply to both, or such that the one would be only a translated versionof the other.

In some embodiments, the updating of the registration of the locatingsystem to the 3D rendering (e.g., CT-base) may be performed in a localmanner and/or in a gradual manner. For example, the registration may beupdated only in the surroundings of the target, e.g., only within acertain distance from the target. This is since the update may be lessaccurate when not performed around the target. In some embodiments, theupdating may be performed in a gradual manner, e.g., by applying weightsaccording to distance from the target. In addition to accuracyconsiderations, such gradual updating may be more convenient or easierfor the user to look at, process and make the necessary changes duringprocedure, than abrupt change in the medical device location on thedisplay.

In some embodiments, the patient may be instructed to stop breathing (orcaused to stop breathing) during the capture of the images in order toprevent movements of the target area due to breathing. In otherembodiments, methods for compensating breathing movements during thecapture of the images may be performed. For example, the estimated posesof the fluoroscopic device may be corrected according to the movementsof a fiducial marker placed in the target area. Such a fiducial may be amedical device, e.g., a catheter, placed in the target area. Themovement of the catheter, for example, may be determined based on thelocating system. In some embodiments, a breathing pattern of the patientmay be determined according to the movements of a fiducial marker, suchas a catheter, located in the target area. The movements may bedetermined via a locating system. Based on that pattern, only images ofinhale or exhale may be considered when determining the pose of theimaging device.

In embodiments, as described above, for each captured frame, the imagingdevice three-dimensional position and orientation are estimated based ona set of static markers positioned on the patient bed. This processrequires knowledge about the markers 3D positions in the volume, as wellas the compatible 2D coordinates of the projections in the image plane.Adding one or more markers from different planes in the volume ofinterest may lead to more robust and accurate pose estimation. Onepossible marker that can be utilized in such a process is the cathetertip (or other medical device tip positioned through the catheter). Thetip is visible throughout the video captured by fluoroscopic imaging andthe compatible 3D positions may be provided by a navigation or trackingsystem (e.g., an electromagnetic navigation tracking system) as the toolis navigated to the target (e.g., through the electromagnetic field).Therefore, the only remaining task is to deduce the exact 2D coordinatesfrom the video frames. As described above, one embodiment of the tipdetection step may include fully automated detection and tracking of thetip throughout the video. Another embodiment may implementsemi-supervised tracking in which the user manually marks the tip in oneor more frames and the detection process computes the tip coordinatesfor the rest of the frames.

In embodiments, the semi-supervised tracking process may be implementedin accordance with solving each frame at a time by template matchingbetween current frame and previous ones, using optical flow to estimatethe tip movement along the video, and/or model-based trackers.Model-based trackers train a detector to estimate the probability ofeach pixel to belong to the catheter tip, which is followed by a step ofcombining the detections to a single most probable list of coordinatesalong the video. One possible embodiment of the model-based trackersinvolves dynamic programming. Such an optimization approach enablesfinding a seam (connected list of coordinates along the video frames 3Dspace—first two dimensions belongs to the image plane and the third axisis time) with maximal probability. Another possible way to achieve aseam of two-dimensional coordinates is training a detector to estimatethe tip coordinate in each frame while incorporating a regularization tothe loss function of proximity between detections in adjacent frames.

FIG. 8 illustrates an exemplary system 800 for constructingfluoroscopic-based three-dimensional volumetric data in accordance withthe disclosure. System 800 may be configured to constructfluoroscopic-based three-dimensional volumetric data of a target areaincluding at least a portion of the lungs of a patient from 2Dfluoroscopic images. System 800 may be further configured to facilitateapproach of a medical device to the target area by using ElectromagneticNavigation Bronchoscopy (ENB) and for determining the location of amedical device with respect to the target.

System 800 may be configured for reviewing CT image data to identify oneor more targets, planning a pathway to an identified target (planningphase), navigating an extended working channel (EWC) 812 of a catheterassembly to a target (navigation phase) via a user interface, andconfirming placement of EWC 812 relative to the target. One such EMNsystem is the ELECTROMAGNETIC NAVIGATION BRONCHOSCOPY® system currentlysold by Medtronic PLC. The target may be tissue of interest identifiedby review of the CT image data during the planning phase. Followingnavigation, a medical device, such as a biopsy tool or other tool, maybe inserted into EWC 812 to obtain a tissue sample from the tissuelocated at, or proximate to, the target.

FIG. 8 illustrates EWC 812 which is part of a catheter guide assembly840. In practice, EWC 812 is inserted into a bronchoscope 830 for accessto a luminal network of the patient “P.” Specifically, EWC 812 ofcatheter guide assembly 840 may be inserted into a working channel ofbronchoscope 830 for navigation through a patient's luminal network. Alocatable guide (LG) 832, including a sensor 844 is inserted into EWC812 and locked into position such that sensor 844 extends a desireddistance beyond the distal tip of EWC 812. The position and orientationof sensor 844 relative to the reference coordinate system, and thus thedistal portion of EWC 812, within an electromagnetic field can bederived. Catheter guide assemblies 840 are currently marketed and soldby Medtronic PLC under the brand names SUPERDIMENSION® Procedure Kits,or EDGE™ Procedure Kits, and are contemplated as useable with thedisclosure. For a more detailed description of catheter guide assemblies840, reference is made to commonly-owned U.S. Patent Publication No.2014/0046315, filed on Mar. 15, 2013, by Ladtkow et al, U.S. Pat. Nos.7,233,820, and 9,044,254, the entire contents of each of which arehereby incorporated by reference.

System 800 generally includes an operating table 820 configured tosupport a patient “P,” a bronchoscope 830 configured for insertionthrough the patient's “P's” mouth into the patient's “P's” airways;monitoring equipment 835 coupled to bronchoscope 830 (e.g., a videodisplay, for displaying the video images received from the video imagingsystem of bronchoscope 830); a locating system 850 including a locatingmodule 852, a plurality of reference sensors 854 and a transmitter matcoupled to a structure of markers 856; and a computing device 825including software and/or hardware used to facilitate identification ofa target, pathway planning to the target, navigation of a medical deviceto the target, and confirmation of placement of EWC 812, or a suitabledevice therethrough, relative to the target. Computing device 825 may besimilar to workstation 80 of FIG. 2A and may be configured, inter alia,to execute the method of FIG. 1.

A fluoroscopic imaging device 810 capable of acquiring fluoroscopic orx-ray images or video of the patient “P” is also included in thisparticular aspect of system 800. The images, sequence of images, orvideo captured by fluoroscopic imaging device 810 may be stored withinfluoroscopic imaging device 810 or transmitted to computing device 825for storage, processing, and display, as described with respect to FIG.2A. Additionally, fluoroscopic imaging device 810 may move relative tothe patient “P” so that images may be acquired from different angles orperspectives relative to patient “P” to create a sequence offluoroscopic images, such as a fluoroscopic video. The pose offluoroscopic imaging device 810 relative to patient “P” and for theimages may be estimated via the structure of markers and according tothe method of FIG. 1. The structure of markers is positioned underpatient “P,” between patient “P” and operating table 820 and betweenpatient “P” and a radiation source of fluoroscopic imaging device 810.Structure of markers is coupled to the transmitter mat (both indicated856) and positioned under patient “P” on operating table 820. Structureof markers and transmitter mat 856 are positioned under the target areawithin the patient in a stationary manner. Structure of markers andtransmitter mat 856 may be two separate elements which may be coupled ina fixed manner or alternatively may be manufactured as one unit.Fluoroscopic imaging device 810 may include a single imaging device ormore than one imaging device. In embodiments including multiple imagingdevices, each imaging device may be a different type of imaging deviceor the same type. Further details regarding the imaging device 810 aredescribed in U.S. Pat. No. 8,565,858, which is incorporated by referencein its entirety herein.

Computing device 185 may be any suitable computing device including aprocessor and storage medium, wherein the processor is capable ofexecuting instructions stored on the storage medium. Computing device185 may further include a database configured to store patient data, CTdata sets including CT images, fluoroscopic data sets includingfluoroscopic images and video, navigation plans, and any other suchdata. Although not explicitly illustrated, computing device 185 mayinclude inputs, or may otherwise be configured to receive, CT data sets,fluoroscopic images/video and other data described herein. Additionally,computing device 185 includes a display configured to display graphicaluser interfaces. Computing device 185 may be connected to one or morenetworks through which one or more databases may be accessed.

With respect to the planning phase, computing device 185 utilizespreviously acquired CT image data for generating and viewing a threedimensional model of the patient's “P's” airways, enables theidentification of a target on the three dimensional model(automatically, semi-automatically, or manually), and allows fordetermining a pathway through the patient's “P's” airways to tissuelocated at and around the target. More specifically, CT images acquiredfrom previous CT scans are processed and assembled into athree-dimensional CT volume, which is then utilized to generate athree-dimensional model of the patient's “P's” airways. Thethree-dimensional model may be displayed on a display associated withcomputing device 185, or in any other suitable fashion. Using computingdevice 185, various views of the three-dimensional model or enhancedtwo-dimensional images generated from the three-dimensional model arepresented. The enhanced two-dimensional images may possess somethree-dimensional capabilities because they are generated fromthree-dimensional data. The three-dimensional model may be manipulatedto facilitate identification of target on the three-dimensional model ortwo-dimensional images, and selection of a suitable pathway through thepatient's “P's” airways to access tissue located at the target can bemade. Once selected, the pathway plan, three dimensional model, andimages derived therefrom, can be saved and exported to a navigationsystem for use during the navigation phase(s). One such planningsoftware is the ILOGIC® planning suite currently sold by Medtronic PLC.

With respect to the navigation phase, a six degrees-of-freedomelectromagnetic locating or tracking system 850, e.g., similar to thosedisclosed in U.S. Pat. Nos. 8,467,589, 6,188,355, and published PCTApplication Nos. WO 00/10456 and WO 01/67035, the entire contents ofeach of which are incorporated herein by reference, or other suitablepositioning measuring system, is utilized for performing registration ofthe images and the pathway for navigation, although other configurationsare also contemplated. Tracking system 850 includes a locating ortracking module 852, a plurality of reference sensors 854, and atransmitter mat 856. Tracking system 850 is configured for use with alocatable guide 832 and particularly sensor 844. As described above,locatable guide 832 and sensor 844 are configured for insertion throughan EWC 182 into a patient's “P's” airways (either with or withoutbronchoscope 830) and are selectively lockable relative to one anothervia a locking mechanism.

Transmitter mat 856 is positioned beneath patient “P.” Transmitter mat856 generates an electromagnetic field around at least a portion of thepatient “P” within which the position of a plurality of referencesensors 854 and the sensor 844 can be determined with use of a trackingmodule 852. One or more of reference sensors 854 are attached to thechest of the patient “P.” The six degrees of freedom coordinates ofreference sensors 854 are sent to computing device 825 (which includesthe appropriate software) where they are used to calculate a patientcoordinate frame of reference. Registration, is generally performed tocoordinate locations of the three-dimensional model and two-dimensionalimages from the planning phase with the patient's “P's” airways asobserved through the bronchoscope 830, and allow for the navigationphase to be undertaken with precise knowledge of the location of thesensor 844, even in portions of the airway where the bronchoscope 830cannot reach. Further details of such a registration technique and theirimplementation in luminal navigation can be found in U.S. PatentApplication Pub. No. 2011/0085720, the entire content of which isincorporated herein by reference, although other suitable techniques arealso contemplated.

Registration of the patient's “P's” location on the transmitter mat 856is performed by moving LG 832 through the airways of the patient's “P.”More specifically, data pertaining to locations of sensor 844, whilelocatable guide 832 is moving through the airways, is recorded usingtransmitter mat 856, reference sensors 854, and tracking module 852. Ashape resulting from this location data is compared to an interiorgeometry of passages of the three dimensional model generated in theplanning phase, and a location correlation between the shape and thethree dimensional model based on the comparison is determined, e.g.,utilizing the software on computing device 825. In addition, thesoftware identifies non-tissue space (e.g., air filled cavities) in thethree-dimensional model. The software aligns, or registers, an imagerepresenting a location of sensor 844 with the three-dimensional modeland two-dimensional images generated from the three-dimension model,which are based on the recorded location data and an assumption thatlocatable guide 832 remains located in non-tissue space in the patient's“P's” airways. Alternatively, a manual registration technique may beemployed by navigating the bronchoscope 830 with the sensor 844 topre-specified locations in the lungs of the patient “P”, and manuallycorrelating the images from the bronchoscope to the model data of thethree dimensional model.

Following registration of the patient “P” to the image data and pathwayplan, a user interface is displayed in the navigation software whichsets for the pathway that the clinician is to follow to reach thetarget. One such navigation software is the ILOGIC® navigation suitecurrently sold by Medtronic PLC.

Once EWC 812 has been successfully navigated proximate the target asdepicted on the user interface, the locatable guide 832 may be unlockedfrom EWC 812 and removed, leaving EWC 812 in place as a guide channelfor guiding medical devices including without limitation, opticalsystems, ultrasound probes, marker placement tools, biopsy tools,ablation tools (i.e., microwave ablation devices), laser probes,cryogenic probes, sensor probes, and aspirating needles to the target.

The disclosed exemplary system 800 may be employed by the method of FIG.7 to construct fluoroscopic-based three-dimensional volumetric data of atarget located in the lungs area and to correct the location of amedical device navigated to the target area with respect to the target.

System 800 or similar version of it in conjunction with the method ofFIG. 7 may be used in various procedures, other than ENB procedures withthe required modifications, and such as laparoscopy or robotic-assistedsurgery.

Systems and methods in accordance with the disclosure may be usable forfacilitating the navigation of a medical device to a target and/or itsarea using real-time two-dimensional fluoroscopic images of the targetarea. The navigation is facilitated by using local three-dimensionalvolumetric data, in which small soft-tissue objects are visible,constructed from a sequence of fluoroscopic images captured by astandard fluoroscopic imaging device available in most procedure rooms.The fluoroscopic-based constructed local three-dimensional volumetricdata may be used to correct a location of a medical device with respectto a target or may be locally registered with previously acquiredvolumetric data (e.g., CT data). In general, the location of the medicaldevice may be determined by a tracking system, for example, anelectromagnetic tracking system. The tracking system may be registeredwith the previously acquired volumetric data. A local registration ofthe real-time three-dimensional fluoroscopic data to the previouslyacquired volumetric data may be then performed via the tracking system.Such real-time data, may be used, for example, for guidance, navigationplanning, improved navigation accuracy, navigation confirmation, andtreatment confirmation.

In some embodiments, the methods disclosed may further include a stepfor generating a 3D rendering of the target area based on apre-operative CT scan. A display of the target area may then include adisplay of the 3D rendering. In another step, the tracking system may beregistered with the 3D rendering. As described above, a correction ofthe location of the medical device with respect to the target, based onthe determined offset, may then include the local updating of theregistration between the tracking system and the 3D rendering in thetarget area. In some embodiments, the methods disclosed may furtherinclude a step for registering the fluoroscopic 3D reconstruction to thetracking system. In another step, and based on the above, a localregistration between the fluoroscopic 3D reconstruction and the 3Drendering may be performed in the target area.

From the foregoing and with reference to the various figure drawings,those skilled in the art will appreciate that certain modifications canalso be made to the disclosure without departing from the scope of thesame. For example, although the systems and methods are described asusable with an EMN system for navigation through a luminal network suchas the lungs, the systems and methods described herein may be utilizedwith systems that utilize other navigation and treatment devices such aspercutaneous devices. Additionally, although the above-described systemand method is described as used within a patient's luminal network, itis appreciated that the above-described systems and methods may beutilized in other target regions such as the liver. Further, theabove-described systems and methods are also usable for transthoracicneedle aspiration procedures.

Detailed embodiments of the disclosure are disclosed herein. However,the disclosed embodiments are merely examples of the disclosure, whichmay be embodied in various forms and aspects. Therefore, specificstructural and functional details disclosed herein are not to beinterpreted as limiting, but merely as a basis for the claims and as arepresentative basis for teaching one skilled in the art to variouslyemploy the disclosure in virtually any appropriately detailed structure.

As can be appreciated a medical instrument such as a biopsy tool or anenergy device, such as a microwave ablation catheter, that ispositionable through one or more branched luminal networks of a patientto treat tissue may prove useful in the surgical arena and thedisclosure is directed to systems and methods that are usable with suchinstruments and tools. Access to luminal networks may be percutaneous orthrough natural orifice using navigation techniques. Additionally,navigation through a luminal network may be accomplished usingimage-guidance. These image-guidance systems may be separate orintegrated with the energy device or a separate access tool and mayinclude MRI, CT, fluoroscopy, ultrasound, electrical impedancetomography, optical, and/or device tracking systems. Methodologies forlocating the access tool include EM, IR, echolocation, optical, andothers. Tracking systems may be integrated to an imaging device, wheretracking is done in virtual space or fused with preoperative or liveimages. In some cases the treatment target may be directly accessed fromwithin the lumen, such as for the treatment of the endobronchial wallfor COPD, Asthma, lung cancer, etc. In other cases, the energy deviceand/or an additional access tool may be required to pierce the lumen andextend into other tissues to reach the target, such as for the treatmentof disease within the parenchyma. Final localization and confirmation ofenergy device or tool placement may be performed with imaging and/ornavigational guidance using a standard fluoroscopic imaging deviceincorporated with methods and systems described above.

While several embodiments of the disclosure have been shown in thedrawings, it is not intended that the disclosure be limited thereto, asit is intended that the disclosure be as broad in scope as the art willallow and that the specification be read likewise. Therefore, the abovedescription should not be construed as limiting, but merely asexemplifications of particular embodiments. Those skilled in the artwill envision other modifications within the scope and spirit of theclaims appended hereto.

What is claimed is:
 1. A system for constructing fluoroscopic-basedthree-dimensional volumetric data of a target area within a patient,comprising: a structure of markers, wherein a sequence of 2Dfluoroscopic images of the target area and of the structure of markersis acquired via a fluoroscopic imaging device; a locating systemindicating a location of a medical device within the patient; and acomputing device configured to: estimate a pose of the fluoroscopicimaging device for each image of a plurality of images of the sequenceof 2D fluoroscopic images based on detecting a projection of thestructure of markers as a whole on each image of the plurality ofimages; construct fluoroscopic-based three-dimensional volumetric dataof the target area from the plurality of images of the sequence of 2Dfluoroscopic images and the estimated poses of the fluoroscopic imagingdevice; cause a display to display the medical device and a 3D renderingof the target area; register the locating system to the 3D rendering;and correct the display of the location of the medical device withrespect to a target by updating the registering of the locating systemto the 3D rendering.
 2. The system of claim 1, wherein the computingdevice is further configured to: facilitate an approach of the medicaldevice to the target area, wherein the medical device is positioned inthe target area prior to acquiring the sequence of 2D fluoroscopicimages; and determine an offset between the medical device and thetarget based on the fluoroscopic-based three-dimensional volumetricdata.
 3. The system of claim 2, wherein the computing device is furtherconfigured to correct the display of the location of the medical devicewith respect to the target based on the offset between the medicaldevice and the target.
 4. The system of claim 3, wherein the target areacomprises at least a portion of a lung and the medical device isnavigable to the target area through airways of a luminal network. 5.The system of claim 1, wherein the locating system is an electromagneticlocating system.
 6. The system of claim 1, wherein the structure ofmarkers is at least one of a periodic pattern or a two-dimensionalpattern.
 7. The system of claim 6, wherein the periodic pattern is a 2Dpattern or a grid pattern.
 8. The system of claim 1, wherein the targetarea comprises at least a portion of a lung and a target is asoft-tissue target.
 9. A method for constructing fluoroscopic-basedthree dimensional volumetric data of a target area within a patient froma sequence of two-dimensional (2D) fluoroscopic images of a target areaand of a structure of markers acquired via a fluoroscopic imagingdevice, wherein the structure of markers is positioned between thepatient and the fluoroscopic imaging device, the method comprising usingat least one hardware processor for: facilitating navigation of amedical device to the target area via a locating system indicating alocation of the medical device and via a display; estimating a pose ofthe fluoroscopic imaging device for each image of at least a pluralityof images of the sequence of 2D fluoroscopic images based on detectionof a projection of the structure of markers on each image of theplurality of images; and constructing fluoroscopic-basedthree-dimensional volumetric data of the target area from the pluralityof images of the sequence of 2D fluoroscopic images and the estimatedposes of the fluoroscopic imaging device; displaying a 3D rendering ofthe target area on the display; registering the locating system to the3D rendering; and correcting of the location of the medical device withrespect to a target by updating the registering of the locating systemto the 3D rendering.
 10. The method of claim 9, wherein a medical deviceis positioned in the target area prior to acquiring the sequence of 2Dfluoroscopic images, and wherein the method further comprises using theat least one hardware processor for determining an offset between themedical device and a target based on the fluoroscopic-basedthree-dimensional volumetric data.
 11. The method of claim 10, furthercomprising using the at least one hardware processor for correcting adisplay of the location of the medical device with respect to the targetbased on the offset between the medical device and the target.
 12. Themethod of claim 11, wherein the structure of markers is at least one ofa periodic pattern or a two-dimensional pattern.
 13. The method of claim12, wherein the periodic pattern is a 2D pattern or a grid pattern. 14.The method of claim 11, wherein the target area comprises at least aportion of lungs and the target is a soft-tissue target.
 15. The methodof claim 10, wherein the target area comprises at least a portion oflungs and wherein the medical device is navigable to the target areathrough airways of a luminal network.
 16. The method of claim 9, furthercomprising using the at least one hardware processor for generating the3D rendering of the target area based on previously acquired CTvolumetric data of the target area.
 17. A system for constructingfluoroscopic-based three-dimensional volumetric data of a target areawithin a patient from two-dimensional (2D) fluoroscopic images acquiredvia a fluoroscopic imaging device, comprising: a computing deviceconfigured to: facilitate an approach of a medical device to the targetarea; estimate a pose of the fluoroscopic imaging device for each imageof a plurality of images of a sequence of 2D fluoroscopic images basedon detection of a projection of a structure of markers on each image ofthe plurality of images; construct fluoroscopic-based three-dimensionalvolumetric data of the target area from the plurality of images of asequence of 2D fluoroscopic images and the estimated poses of thefluoroscopic imaging device; display the medical device and a 3Drendering of the target area on the display; register a locating systemindicating a location of the medical device within the patient to the 3Drendering; and correct the display of the location of the medical devicewith respect to a target comprises updating the registering of thelocating system to the 3D rendering.
 18. The system of claim 17, whereinthe medical device is positioned in the target area prior to acquisitionof the sequence of 2D fluoroscopic images, and wherein the computingdevice is further configured to determine an offset between the medicaldevice and the target based on the fluoroscopic-based three-dimensionalvolumetric data.
 19. The system of claim 18, further comprising alocating system indicating the location of the medical device within thepatient, wherein the computing device comprises a display and isconfigured to: display the target area and the location of the medicaldevice with respect to the target; facilitate navigation of the medicaldevice to the target area via the locating system and the display; andcorrect the display of the location of the medical device with respectto the target based on the offset between the medical device and thetarget.
 20. The system of claim 17, wherein the structure of markers isat least one of a periodic pattern, a 2D pattern, or a grid pattern.